{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n",
    "预测值就会比训练值大很多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.300479089768492\n",
      "W1 relative error: 1.03e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 4.56e-07\n",
      "b1 relative error: 4.66e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 1.69e-10\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.85e-07\n",
      "W2 relative error: 2.15e-06\n",
      "W3 relative error: 4.56e-08\n",
      "b1 relative error: 1.16e-08\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.48e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.30427592207859\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 2.48e-08\n",
      "W3 relative error: 6.43e-08\n",
      "b1 relative error: 5.37e-09\n",
      "b2 relative error: 1.91e-09\n",
      "b3 relative error: 1.85e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.888000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.926000; val_acc: 0.277000\n",
      "(Epoch 12 / 25) train acc: 0.962000; val_acc: 0.302000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.307000\n",
      "(Epoch 14 / 25) train acc: 0.966000; val_acc: 0.309000\n",
      "(Epoch 15 / 25) train acc: 0.978000; val_acc: 0.291000\n",
      "(Epoch 16 / 25) train acc: 0.986000; val_acc: 0.302000\n",
      "(Epoch 17 / 25) train acc: 0.986000; val_acc: 0.315000\n",
      "(Epoch 18 / 25) train acc: 0.996000; val_acc: 0.319000\n",
      "(Epoch 19 / 25) train acc: 0.992000; val_acc: 0.312000\n",
      "(Epoch 20 / 25) train acc: 0.984000; val_acc: 0.315000\n",
      "(Iteration 101 / 125) loss: 0.129474\n",
      "(Epoch 21 / 25) train acc: 0.994000; val_acc: 0.315000\n",
      "(Epoch 22 / 25) train acc: 0.970000; val_acc: 0.317000\n",
      "(Epoch 23 / 25) train acc: 0.984000; val_acc: 0.308000\n",
      "(Epoch 24 / 25) train acc: 0.982000; val_acc: 0.301000\n",
      "(Epoch 25 / 25) train acc: 0.992000; val_acc: 0.303000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318478\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.626000; val_acc: 0.290000\n",
      "(Epoch 7 / 25) train acc: 0.628000; val_acc: 0.298000\n",
      "(Epoch 8 / 25) train acc: 0.686000; val_acc: 0.310000\n",
      "(Epoch 9 / 25) train acc: 0.722000; val_acc: 0.289000\n",
      "(Epoch 10 / 25) train acc: 0.724000; val_acc: 0.300000\n",
      "(Epoch 11 / 25) train acc: 0.760000; val_acc: 0.305000\n",
      "(Epoch 12 / 25) train acc: 0.772000; val_acc: 0.280000\n",
      "(Epoch 13 / 25) train acc: 0.814000; val_acc: 0.303000\n",
      "(Epoch 14 / 25) train acc: 0.814000; val_acc: 0.341000\n",
      "(Epoch 15 / 25) train acc: 0.856000; val_acc: 0.352000\n",
      "(Epoch 16 / 25) train acc: 0.840000; val_acc: 0.302000\n",
      "(Epoch 17 / 25) train acc: 0.840000; val_acc: 0.291000\n",
      "(Epoch 18 / 25) train acc: 0.846000; val_acc: 0.316000\n",
      "(Epoch 19 / 25) train acc: 0.868000; val_acc: 0.326000\n",
      "(Epoch 20 / 25) train acc: 0.860000; val_acc: 0.311000\n",
      "(Iteration 101 / 125) loss: 5.299135\n",
      "(Epoch 21 / 25) train acc: 0.890000; val_acc: 0.318000\n",
      "(Epoch 22 / 25) train acc: 0.876000; val_acc: 0.300000\n",
      "(Epoch 23 / 25) train acc: 0.908000; val_acc: 0.312000\n",
      "(Epoch 24 / 25) train acc: 0.894000; val_acc: 0.315000\n",
      "(Epoch 25 / 25) train acc: 0.922000; val_acc: 0.323000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xt83HWZ6PHPYwkSFRsuFSEttEi3gG2hGIoLy0W6WBApBbVSFRd3FVERj+5W4ZzdntqzrqxF8bCw66k3wAvQZWstClYFb+iKpBZbKVYKgk1ALZdU0IClPOePmZRpTNqkzcxvMvm8X6+8Zn7P7ze/7zPNEPLke4vMRJIkSZKk5xWdgCRJkiSpPlggSpIkSZIAC0RJkiRJUpkFoiRJkiQJsECUJEmSJJVZIEqSJEmSAAtESZKIiFER8WREHFh0LpIkFSncB1GSNNxExJMVhy8Anga2lI/fmZlfqn1WkiQNfxaIkqRhLSIeAN6emd/ezjW7ZeYztcuqdhr5vUmSas8hppKkhhMR/xwRN0TEdRHxBPCWiPjLiPhxRHRFxMMRcUVENJWv3y0iMiLGl4+/WD5/S0Q8ERH/HRET+mnreRFxY0T8pnzv70bEYRXnXxARl0fEryNiU0R8PyKeXz53QjmnTRGxISLOLcdvj4jzKu7x9oj4bq9c3x0R64FflONXRkRHRPw+Iu6MiGMrXr9bRPxTRNxXPt8eEQdExP+LiH/t9X5ujoj37vp3QZI0HFkgSpIa1VnAl4HRwA3AM8D7gH2B44BTgXdu5/VvAv4J2Bv4NfB/tnPt14CJwEuBnwNfqDh3OTAVOKZ8r/8JPFsuOG8GPgHsA0wD1gzi/c0CjgamlI/vKLezN3Aj8J89hSgwD3g9pffcArwdeAq4BpgbEQEQEfsBJwHXDSIPSVIDsUCUJDWq2zPzpsx8NjO7M/POzLwjM5/JzPuBxcCJ23n9jZnZnpmbgS8BR/Z1Ufn+V2fmE5n5FLAAeEVEvDAiRgHnARdl5sOZuSUzby/f8y3ALZm5pJzTI5l51yDe379k5uOZ2V3O4wuZ+Vh5uOnHgBcDh5SvfTvwPzPz3nK+d5Wv/RGlQvGk8nVzgW9n5iODyEOS1EAsECVJjWpD5UFEHBoRXy8PBf09sJBSb2J/flPx/I/Ai/q6qLwC6sci4v7yfdeXT+0L7AfsDtzXx0vH9RMfqN7v74MR8YuI2AQ8DryQ597f9tq6llKxSvnxC/1cJ0kaASwQJUmNqvcqbP+P0vDPQzLzxcB8IIagnbcCrwFOpjSctafXLoDfAn8CXtbH6zb0Ewf4A6XVWXu8tI9rtr6/iHgV8AHgdZSGkO4FPMlz7297bX0BOCsippWvuamf6yRJI4AFoiRppNgT2AT8obyIzPbmHw72vk8Dj1Iq6j7ScyIztwBXA5+MiJeWexuPKy+O80Xg1Ih4XXkRmX0j4ojyS+8CXhcRzRHxF8DfDiCHZ4BHgCZKw1xfWHH+M8A/R8TLouTIiNi7nOODwM8ozUf8z/IwWUnSCGWBKEkaKf4e+BvgCUq9iTcM0X0/DzxU/rob+FGv8+8H7gFWAo8B/0Jpm6lfAWcAHyrHf8pzC85cRqmH8HfA5ygVk9tzM/Bt4F7gAeD3wMMV5xcBy4Bby+cWA3tUnL+m3LbDSyVphHMfREmSRriIOBn4LHBw+ouBJI1o9iBKkjSCRcTulLb/+LTFoSTJAlGSpBEqIqZQWvF0b+CKgtORJNUBh5hKkiRJkgB7ECVJkiRJZRaIkiRJkiQAdis6gVrYd999c/z48UWnIUmSJEmFWLly5SOZOWZH142IAnH8+PG0t7cXnYYkSZIkFSIiHhzIdQ4xlSRJkiQBFoiSJEmSpDILREmSJEkSYIEoSZIkSSqrqwIxIj4XEb+LiJ/3cz4i4oqIWB8RqyPiqFrnKEmSJEmNqq4KROBq4NTtnD8NmFj+Oh/4jxrkJEmSJEkjQl1tc5GZ34+I8du55Ezg2sxM4McR0RIR+2fmwzVJUJIkqUEsW9XJohXreKirmwNampk3cxKzp7UWnZakgtVVgTgArcCGiuOOcswCUZIkaYCWrerkkqVr6N68BYDOrm4uWboGoOpF4kgsTEfiex6JGuX7XG9DTIdMRJwfEe0R0b5x48ai05EkSaobi1as21oc9ujevIVFK9ZVtd2ewrSzq5vkucJ02arOqrZbpJH4nkeiRvo+D7cCsRMYV3E8thz7M5m5ODPbMrNtzJgxNUlOkiQNT8tWdXLcpbcx4eKvc9ylt9Xsl7qi2n2oq3tQ8aFSVGFapJH4nkeiRvo+D7chpsuBCyPieuAYYJPzDyVJ0q4oarhlkcM8D2hpprOPYvCAluaqtltUYdqjiCGARb/nohQ13LKodhvp+1xXPYgRcR3w38CkiOiIiL+LiAsi4oLyJTcD9wPrgU8D7y4oVUmSNMSK6k0r6i//RfY4zJs5ieamUdvEmptGMW/mpKq2218BWu3CFIobAljkey5KUf/WRQ7zbKTvc10ViJk5NzP3z8ymzBybmZ/NzE9l5qfK5zMz35OZL8vMKZnZXnTOkiRp1xX5i11Rf/kvssdh9rRWPnr2FFpbmgmgtaWZj549peo9LUUVplBcQV7key6Kf3QpGa7f5+E2xFSSJDWg7f1i16jDLYtqt8fsaa01X2Gxp72RNASwyPc80oZbFv1HFyjm+zzULBAlSVLhivzFbt7MSdvMBYTa/OW/qHaLVkRhCsUW5EW855E4x3Uk/tGlGupqiKkkSXrOSFpZs8j5O0UNtyyq3ZGqkYYADsRIHG450r7H1WIPoiRJ21HkSnwjaWXNonvTivrLf6P0OAwHjTQEcCBG4nDLkfY9rpbIzKJzqLq2trZsb3c9G0nS4PQulqBUtNSil+e4S2/rc6hUa0szP7z45IZrF4orxqVGVOR/y6pPEbEyM9t2dJ09iJIk9aPIhVNG6iIPFoTS0Ci6V17Dl3MQJUnqR5HFUlFz8hppLy9pJHOOq3aWPYiSJPWjyBXxXFlT0q6yV147wx5ESZL6UeSKeK6sKUkqgovUSJK0HS6cIklqBC5SI0nSEHCIliRpJHGIqSRJkiQJsECUJEmSJJU5xFSSNCw4F1CSpOqzQJQk1b1lqzq32Xqhs6ubS5auAbBIlCTVh9VL4NaFsKkDRo+FGfNh6pyisxo0h5hKkureohXrttmXD6B78xYWrVhXUEaSJFVYvQRuugg2bQCy9HjTRaX4MGOBKEmqew/1sVn99uKSJNXUrQthc6//J23uLsWHGQtESVLdO6CleVBxSZJqalPH4OJ1zAJRklT35s2cRHPTqG1izU2jmDdzUkEZSZK2a/USuHwyLGgpPQ7DoZaDMnrs4OJ1rO4KxIg4NSLWRcT6iLi4j/MHRcStEbE6Ir4bEcPvX12SNCizp7Xy0bOn0NrSTACtLc189OwpLlAjSfWogebjDdiM+dDUa1RLU3MpPsxEZhadw1YRMQr4JXAK0AHcCczNzLUV1/wn8LXMvCYiTgbelpnnbu++bW1t2d7eXsXMJam23PJBklS3Lp9cLg57GT0O3v/z2udTK3W+imlErMzMth1dV2/bXEwH1mfm/QARcT1wJrC24prDgQ+Un38HWFbTDCWpYG75IEmqaw00H29Qps6pq4JwZ9XbENNWoPLPDR3lWKWfAWeXn58F7BkR+9QgN0mqC275IEmqa0XOxxtpcx+roN4KxIH4B+DEiFgFnAh0Alt6XxQR50dEe0S0b9y4sdY5SlLVuOWDJA0zI61oKWo+3kic+1gF9VYgdgLjKo7HlmNbZeZDmXl2Zk4D/lc51tX7Rpm5ODPbMrNtzJgx1cxZkmqqyC0flq3q5LhLb2PCxV/nuEtvY9mqzh2/SJJGspFYtEydA2dcUZpzSJQez7ii+sMvG2gvwiLV2xzEO4GJETGBUmF4DvCmygsiYl/gscx8FrgE+FzNs5SkAs2bOWmbOYhQmy0fnPsoSTthe0VLA8xX61cR8/FG6tzHIVZXPYiZ+QxwIbACuAdYkpl3R8TCiJhVvuwkYF1E/BLYD/hIIclKUkGK2vLBuY+StBMsWmqngfYiLFK99SCSmTcDN/eKza94fiNwY63zkqTeitxqYva01pr32jn3UZJ2wuix/Wz5UKMFW+p424UhN2N+afhuZY/tMN2LsEh11YMoScNFz3DLzq5ukueGWzbynLwi5z5K0rDlgi21U9TcxwZjgShJO2EkDrecN3MSzU2jtonVYu6jJA1rLthSW1PnwPt/Dgu6So8Wh4NWd0NMJWk4GInDLXuGtBY1rFZSgxhpwx7BBVs0rFggStJOOKClmc4+isFGH25ZxNxHSQ2kZ9hjT89Wz7BHaPwisdaKnPuoYc0hppK0ExxuKUk7oehhjyNpw/qi5j5q2LMHUZJ2gsMtpSoYiUMPR5oihz2OtN7Lnvfkf1MapMjMonOoura2tmxvby86DUmS1J/ev7xDqbfDFQgby+WT+xn2OK60oEijti3VgYhYmZltO7rOIaaSJKl4RQ89VG0UOezRRVukAbFAlCRJxfOX99orYj5ekfvU9bc4i4u2SNtwDqIkSSpe0SsuFjX/sch2i5qPV8SWD1D6t+1rGLOLtkjbsAdRkiQVr8ihhz3F0qYNQD5XLFW7R62odmFkDuktsvdSGkbsQZQkScUrcsXF7RVL1Wy/qHZh5A7pLar3UhpGLBAlDXvLVnW63YTUCIr65b2oYqnIIq3oIb2S6pZDTCUNa8tWdXLJ0jV0dnWTQGdXN5csXcOyVZ1FpyZpuChq8ZIiF01xE3VJ/bBAlDSsLVqxju7NW7aJdW/ewqIV6wrKSNKwU1SxVGSR5nw8Sf1wiKmkYe2hru5BxSXpzxQ1/7HIeZc97VsQSurFAlHSsHZASzOdfRSDB7Q093G1JPWjqGLJIk1SnXGIqaRhbd7MSTQ3jdom1tw0inkzJxWUkTSEitjIvMh2JUmFswdR0rDWs1qpq5iq4RS1kXmRG6hLkgoXmVl0DlXX1taW7e3tRachSdLAXT65n20IxsH7f9547UqSqioiVmZm246uq7shphFxakSsi4j1EXFxH+cPjIjvRMSqiFgdEa8pIk9JkqpqJO7NJ0kqXF0ViBExCrgKOA04HJgbEYf3uuwfgSWZOQ04B/j32mYpSVINjMS9+SRJhaurAhGYDqzPzPsz80/A9cCZva5J4MXl56OBh2qYnyRJtTES9+aTJBWu3grEVqBy4kNHOVZpAfCWiOgAbgbeW5vUJEmqoaI2MncDdUka0YbjKqZzgasz8+MR8ZfAFyJicmY+W3lRRJwPnA9w4IEHFpCmJEm7yL35JEk1Vm89iJ3AuIrjseVYpb8DlgBk5n8DewD79r5RZi7OzLbMbBszZkyV0pUkSZKkxlFvBeKdwMSImBARu1NahGZ5r2t+DcwAiIjDKBWIG2uapSRJkiQ1oLoqEDPzGeBCYAVwD6XVSu+OiIURMat82d8D74iInwHXAeflSNjMUZIkSZKqrO7mIGbmzZQWn6mMza94vhY4rtZ5SZIkSVKjq6seREmS6s7qJXD5ZFjQUnpcvaTojCRJqpq660GUJKlurF4CN10Em7tLx5s2lI7BVT4lSQ3JHkRJkvpz68LnisMem7tLcUmSGpAFoiRpeChiqOemjsHFJUka5iwQJUn1r2eo56YNQD431LPaReLosYOLS5I0zFkgSpLqX1FDPWfMh6bmbWNNzaW4JEkNyAJRklT/ihrqOXUOnHEFjB4HROnxjCtcoEaS1LBcxVSSVP9Gjy0PL+0jXm1T51gQSpJGDHsQJUn1z6GekiTVhAWiJKn+OdRTkqSacIipJGl4cKinJElVZw+iJEmSJAmwQJQkSZIklVkgSpIkSZIA5yBKGiLLVnWyaMU6Hurq5oCWZubNnMTsaa1Fp9W4Vi8pbRK/qaO01cOM+bWZn1dUu5IkqSYsECXtsmWrOrlk6Rq6N28BoLOrm0uWrgGwSKyG1Uvgpotgc3fpeNOG0jFUt1grql1JklQzDjGVtMsWrVi3tTjs0b15C4tWrCsoowZ368LnirQem7tL8UZsV5Ik1YwFoqRd9lBX96Di2kWbOgYXH+7tSpKkmnGIqdRgipgLeEBLM519FIMHtDRXtd0Ra/TY0vDOvuKN2K4kSaqZuutBjIhTI2JdRKyPiIv7OH95RNxV/vplRHQVkadUj3rmAnZ2dZM8Nxdw2arOqrY7b+YkmptGbRNrbhrFvJmTqtruiDVjPjT1Kr6bmkvxRmxXkiTVTF0ViBExCrgKOA04HJgbEYdXXpOZ78/MIzPzSODfgKW1z1SqT0XNBZw9rZWPnj2F1pZmAmhtaeajZ09xgZpqmToHzrgCRo8DovR4xhXVXyimqHYlSVLN1NsQ0+nA+sy8HyAirgfOBNb2c/1c4H/XKDep7hU5F3D2tFYLwlqaOqeYwqyodiVJUk3UVQ8i0ApUTnDpKMf+TEQcBEwAbqtBXtKw0N+cP+cCSpIkaSDqrUAcjHOAGzNzS18nI+L8iGiPiPaNGzfWODWpGM4FlCRJ0q6otwKxExhXcTy2HOvLOcB1/d0oMxdnZltmto0ZM2YIU5Tql3MBa2z1Erh8MixoKT2uXlJ0RpIkSbuk3uYg3glMjIgJlArDc4A39b4oIg4F9gL+u7bpSfXPuYA1snoJ3HTRcxvHb9pQOgbn6EmSpGGrrnoQM/MZ4EJgBXAPsCQz746IhRExq+LSc4DrMzOLyFOSuHXhc8Vhj83dpbgkSdIwVZUexIh4L/DFzHx8sK/NzJuBm3vF5vc6XrBLCUrSrtrUMbi4JEnSMFCtHsT9gDsjYkl54/uoUjuSVIzRYwcXlyRJGgaqUiBm5j8CE4HPAucB90bEv0TEy6rRniTV3Iz50NRr+5Cm5lJckiRpmKraHMTy/MDflL+eobSozI0R8bFqtSlJNTN1DpxxBYweB0Tp8YwrXKBGkiQNa9Wag/g+4K3AI8BngHmZuTkingfcC3ywGu1KUk1NnWNBKEmSGkq1trnYGzg7Mx+sDGbmsxHx2iq1KUmSJEnaBdUaYnoL8FjPQUS8OCKOAcjMe6rUpiRJkiRpF1SrQPwP4MmK4yfLMUmSJElSnapWgRiVm9hn5rNUbzirJEmSJGkIVKtAvD8iLoqIpvLX+4D7q9SWJEmSJGkIVKtAvAA4FugEOoBjgPOr1JYkSZIkaQhUZdhnZv4OOKca95YkSZIkVUe19kHcA/g74OXAHj3xzPzbarQnSZIkSdp11Rpi+gXgpcBM4HvAWOCJKrUlSZIkSRoC1SoQD8nMfwL+kJnXAKdTmocoqVGtXgKXT4YFLaXH1UtGRtuSJEkNpFpbT2wuP3ZFxGTgN8BLqtSWpKKtXgI3XQSbu0vHmzaUjgGmzmnctiVJkhpMtXoQF0fEXsA/AsuBtcC/VqktSUW7deFzBVqPzd2leCO3LUmS1GCGvAcxIp4H/D4zHwe+Dxw81G1IqjObOgYXb5S2JUmSGsyQ9yBm5rPAB4f6vpLq2Oixg4s3StuSJEkNplpDTL8dEf8QEeMiYu+eryq1JaloM+ZDU/O2sabmUryR25YkSWow1Vqk5o3lx/dUxBKHm0qNqWcxmFsXloZ2jh5bKtBqsUhMkW1LkiQ1mMjMonPYRkScCvxfYBTwmcy8tI9r5gALKBWdP8vMN23vnm1tbdne3l6FbCVJkiSp/kXEysxs29F1VelBjIi39hXPzGt38LpRwFXAKUAHcGdELM/MtRXXTAQuAY7LzMcjwu0zVJeWrepk0Yp1PNTVzQEtzcybOYnZ01qLTkuSJEnqV7WGmB5d8XwPYAbwU2C7BSIwHVifmfcDRMT1wJmUtsno8Q7gqvIqqWTm74YqaWmoLFvVySVL19C9eQsAnV3dXLJ0DYBFoiRJkupWVQrEzHxv5XFEtADXD+ClrcCGiuMO4Jhe1/xF+Z4/pDQMdUFmfmPns5WG3qIV67YWhz26N29h0Yp1FoiSJEmqW9XqQeztD8CEIbrXbsBE4CRgLPD9iJiSmV2VF0XE+cD5AAceeOAQNS0NzENd3YOKS5IkSfWgWnMQb6K0gAyUttI4HFgygJd2AuMqjseWY5U6gDsyczPwq4j4JaWC8c7KizJzMbAYSovUDPY9SLvigJZmOvsoBg9oae7jakmSJKk+VKsH8bKK588AD2ZmxwBedycwMSImUCoMzwF6r1C6DJgLfD4i9qU05PT+XU9ZGjrzZk7aZg4iQHPTKObNnFRgVpIkSdL2VatA/DXwcGY+BRARzRExPjMf2N6LMvOZiLgQWEFpfuHnMvPuiFgItGfm8vK5V0fEWmALMC8zH63S+5B2Ss88Q1cxlSRJ0nBSlX0QI6IdODYz/1Q+3h34YWYevf1XVof7IEqSJEkayQa6D+LzqtT+bj3FIUD5+e5VakuSJEmSNASqVSBujIhZPQcRcSbwSJXakiRJkiQNgWrNQbwA+FJEXFk+7gDeWqW2JEmSJElDoCoFYmbeB7wyIl5UPn6yGu1I6sPqJXDrQtjUAaPHwoz5MHVO0VlJkiRpGKjKENOI+JeIaMnMJzPzyYjYKyL+uRptSaqwegncdBFs2gBk6fGmi0pxSZIkaQeqNQfxtMzs6jnIzMeB11SpLUk9bl0Im7u3jW3uLsUlSZKkHahWgTgqIp7fcxARzcDzt3O9pKGwqWNwcUmSJKlCtRap+RJwa0R8HgjgPOCaKrUlqcfoseXhpX3EJUmSpB2oSg9iZv4r8M/AYcAkYAVwUDXaklRhxnxoat421tRcikuSJEk7UK0hpgC/BRJ4A3AycE8V25IEpdVKz7gCRo8DovR4xhWuYipJkqQBGdIhphHxF8Dc8tcjwA1AZOarhrIdSdsxdY4FoSRJknbKUM9B/AXwA+C1mbkeICLeP8RtSAO2bFUni1as46Gubg5oaWbezEnMntZadFqSJElSXRrqIaZnAw8D34mIT0fEDEqL1Eg1t2xVJ5csXUNnVzcJdHZ1c8nSNSxb1VmbBFYvgcsnw4KW0qN7EUqSJKnODWmBmJnLMvMc4FDgO8D/AF4SEf8REa8eyrakHVm0Yh3dm7dsE+vevIVFK9ZVv3E3rJckSdIwVK1VTP+QmV/OzDOAscAq4EPVaEvqz0Nd3YOKDyk3rJckSdIwVM1VTAHIzMczc3Fmzqh2W1KlA1qaBxUfUm5YL0mSpGGo6gWiVJR5MyfR3DRqm1hz0yjmzZxU/cb725jeDeslSZJUxywQ1bBmT2vlo2dPobWlmQBaW5r56NlTarOKqRvWS5IkaRga6m0upLoye1prMdta9OxDeOvC0rDS0WNLxaH7E0qSJKmO1V2BGBGnAv8XGAV8JjMv7XX+PGAR0LNXwZWZ+ZmaJikNhBvWS5IkaZipqwIxIkYBVwGnAB3AnRGxPDPX9rr0hsy8sOYJavhZvcRePEmSJGmA6m0O4nRgfWben5l/Aq4Hziw4Jw1X7kUoSZIkDUq9FYitwIaK445yrLfXRcTqiLgxIsbVJjUNO+5FKEmSJA1KvRWIA3ETMD4zpwLfAq7p66KIOD8i2iOifePGjTVNUHXCvQglSZKkQam3ArETqOwRHMtzi9EAkJmPZubT5cPPAK/o60aZuTgz2zKzbcyYMVVJVnXOvQglSZKkQam3AvFOYGJETIiI3YFzgOWVF0TE/hWHs4B7apifdtKyVZ0cd+ltTLj46xx36W0sW9W54xftKvcilCRJkgalrlYxzcxnIuJCYAWlbS4+l5l3R8RCoD0zlwMXRcQs4BngMeC8whLWgCxb1cklS9fQvXkLAJ1d3VyydA1AdfcodC9CSZIkaVAiM4vOoera2tqyvb296DRGrOMuvY3Oru4/i7e2NPPDi08uICNJkiRpZImIlZnZtqPr6m2IqRrQQ30Uh9uLS5IkSSqGBaKq7oCW5kHFJUmSJBXDAlFVN2/mJJqbRm0Ta24axbyZkwrKSJIkSVJf6mqRGjWmnoVoFq1Yx0Nd3RzQ0sy8mZOqu0CNJEmSpEGzQFRNzJ7WakEoSZIk1TmHmEqSJEmSAAtESZIkSVKZBaIkSZIkCbBAlCRJkiSVWSBKkiRJkgALREmSJElSmQWiJEmSJAmwQJQkSZIklVkgSpIkSZIAC0RJkiRJUpkFoiRJkiQJgN2KTkC1s2xVJ4tWrOOhrm4OaGlm3sxJzJ7WWnRakiRJkuqEBeIIsWxVJ5csXUP35i0AdHZ1c8nSNQAWiZIkSZIAh5iOGItWrNtaHPbo3ryFRSvWFZSRJEmSpHpjgThCPNTVPai4JEmSpJGn7grEiDg1ItZFxPqIuHg7170uIjIi2mqZ33B1QEvzoOJDbvUSuHwyLGgpPa5eUpt2JUmSJA1YXRWIETEKuAo4DTgcmBsRh/dx3Z7A+4A7apvh8DVv5iSam0ZtE2tuGsW8mZOq3/jqJXDTRbBpA5Clx5suskiUJEmS6kxdFYjAdGB9Zt6fmX8CrgfO7OO6/wP8K/BULZMbzmZPa+WjZ0+htaWZAFpbmvno2VNqs0DNrQthc6+hrJu7S3FJkiRJdaPeVjFtBTZUHHcAx1ReEBFHAeMy8+sRMa+WyQ13s6e1FrNi6aaOwcUlSZIkFaLeehC3KyKeB3wC+PsBXHt+RLRHRPvGjRurn5z6N3rs4OKSJEmSClFvBWInMK7ieGw51mNPYDLw3Yh4AHglsLyvhWoyc3FmtmVm25gxY6qYsnZoxnxo6rUYTlNzKS5JkiSpbtRbgXgnMDEiJkTE7sA5wPKek5m5KTP3zczxmTke+DEwKzPbi0lXAzJ1DpxxBYweB0Tp8YwrSnFJkiRJdaOu5iBm5jMRcSGwAhgFfC4z746IhUB7Zi7f/h20XauXlBaG2dRRGt45Y37tirSpcywIJUmSpDpXVwUiQGbeDNzcK9bnWMTMPKkWOTWEnq0melYT7dlqAizcJEmSJAH1N8RU1eJWE5IkSZJ2wAJxpHCrCUmSJEk7YIE4UrjVhCRJkqQdsEAcKdxqQpIkSdIOWCCOFG41IUmSJGkH6m4VU1WRW01IkiRJ2g57ECVJkiRJgAWiJEmSJKnMAlGSJEmSBFggSpIkSZLKLBAlSZIkSYAFoiRJkiSpzAJRkiRJkgTLcXZSAAAgAElEQVRYIEqSJEmSyiwQJUmSJEmABaIkSZIkqcwCUZIkSZIEWCBKkiRJksosECVJkiRJQB0WiBFxakSsi4j1EXFxH+cviIg1EXFXRNweEYcXkackSZIkNZq6KhAjYhRwFXAacDgwt48C8MuZOSUzjwQ+BnyixmlKkiRJUkOqqwIRmA6sz8z7M/NPwPXAmZUXZObvKw5fCGQN85MkSZKkhrVb0Qn00gpsqDjuAI7pfVFEvAf4ALA7cHJtUpMkSZKkxlZvPYgDkplXZebLgA8B/9jXNRFxfkS0R0T7xo0ba5ugJEmSJA1D9VYgdgLjKo7HlmP9uR6Y3deJzFycmW2Z2TZmzJghTFGSJEmSGlO9FYh3AhMjYkJE7A6cAyyvvCAiJlYcng7cW8P8JEmSJKlh1dUcxMx8JiIuBFYAo4DPZebdEbEQaM/M5cCFEfHXwGbgceBvistYkiRJkhpHXRWIAJl5M3Bzr9j8iufvq3lSkiRJkjQC1NsQU0mSJElSQSwQJUmSJEmABaIkSZIkqazu5iBKkiRpZNm8eTMdHR089dRTRaciDXt77LEHY8eOpampaadeb4FYgGWrOlm0Yh0PdXVzQEsz82ZOYva01qLTkiRJKkRHRwd77rkn48ePJyKKTkcatjKTRx99lI6ODiZMmLBT93CIaY0tW9XJJUvX0NnVTQKdXd1csnQNy1Z1Fp2aJElSIZ566in22Wcfi0NpF0UE++yzzy71xlsg1tiiFevo3rxlm1j35i0sWrGuoIwkSZKKZ3EoDY1d/W/JArHGHurqHlRckiRJ1fe3f/u3vOQlL2Hy5Mn9XpOZXHTRRRxyyCFMnTqVn/70p1vPXXPNNUycOJGJEydyzTXX7LC97373u7z2ta8dktx3xtVXX81DDz1UWPtD7Rvf+AaTJk3ikEMO4dJLL+3zmk984hMcfvjhTJ06lRkzZvDggw9uPTdq1CiOPPJIjjzySGbNmrXD9h544IHtflaqbdmyZaxdu7Yq97ZArLEDWpoHFZckSVL1nXfeeXzjG9/Y7jW33HIL9957L/feey+LFy/mXe96FwCPPfYYH/7wh7njjjv4yU9+woc//GEef/zxncrjmWee2anXDVYjFYhbtmzhPe95D7fccgtr167luuuu67N4mjZtGu3t7axevZrXv/71fPCDH9x6rrm5mbvuuou77rqL5cuX73Qutfr+WSA2kHkzJ9HcNGqbWHPTKObNnFRQRpIkScPLslWdHHfpbUy4+Oscd+ltQ7KWwwknnMDee++93Wu++tWv8ta3vpWI4JWvfCVdXV08/PDDrFixglNOOYW9996bvfbai1NOOaXPYvMb3/gGhx56KEcddRRLly7dGl+wYAHnnnsuxx13HOeeey5PPfUUb3vb25gyZQrTpk3jO9/5DlAq6s4880xOOukkJk6cyIc//OGt9/jEJz7B5MmTmTx5Mp/85CeBP+/luuyyy1iwYAE33ngj7e3tvPnNb+bII4+ku7vGI9lWL4HLJ8OCltLj6iW7dLuf/OQnHHLIIRx88MHsvvvunHPOOXz1q1/9s+te9apX8YIXvACAV77ylXR0dAyqnZUrV3LEEUdwxBFHcNVVV22NX3311cyaNYuTTz6ZGTNmkJnMmzePyZMnM2XKFG644Qag1Gt8wgkncPrppzNp0iQuuOACnn32WQCuu+46pkyZwuTJk/nQhz609d4vetGLtj6/8cYbOe+88/jRj37E8uXLmTdvHkceeST33XffoN7HjriKaY31rFbqKqaSJEmD17PgX8+aDj0L/gFV/32qs7OTcePGbT0eO3YsnZ2d/cYrPfXUU7zjHe/gtttu45BDDuGNb3zjNufXrl3L7bffTnNzMx//+MeJCNasWcMvfvELXv3qV/PLX/4SKBVDP//5z3nBC17A0Ucfzemnn05E8PnPf5477riDzOSYY47hxBNPZK+99urzfbz+9a/nyiuv5LLLLqOtrW2o/nkGZvUSuOki2FwuSjdtKB0DTJ2zU7fs69//jjvu2O5rPvvZz3LaaadtPX7qqadoa2tjt9124+KLL2b27Nl/9pq3ve1tXHnllZxwwgnMmzdvm3M//elPWb16NXvvvTf/9V//xV133cXPfvYzHnnkEY4++mhOOOEEoPT9W7t2LQcddBCnnnoqS5cu5dhjj+VDH/oQK1euZK+99uLVr341y5Yt6zMHgGOPPZZZs2bx2te+lte//vUD/ncaKAvEAsye1mpBKEmStBO2t+BfPf9+9Ytf/IIJEyYwceJEAN7ylrewePHirednzZpFc3NpytHtt9/Oe9/7XgAOPfRQDjrooK0F4imnnMI+++wDwNlnn83tt99ORHDWWWfxwhe+cGv8Bz/4wYDm0tXcrQufKw57bO4uxXeyQBysL37xi7S3t/O9731va+zBBx+ktbWV+++/n5NPPpkpU6bwspe9bOv5rq4uurq6thZ65557LrfccsvW8z09yFD6/s2dO5dRo0ax3377ceKJJ3LnnXfy4he/mOnTp3PwwQcDMHfuXG6//Xaampo46aSTGDNmDABvfvOb+f73v99vgVhtDjEtwhB3q0uSJI0URS7419rayoYNG7Yed3R00Nra2m98MHqKux3pvULl9las3G233bYOYQR2aeuDIbOpn2Gd/cUHYDD//t/+9rf5yEc+wvLly3n+85+/zT0ADj74YE466SRWrVo1qByq8f3rfb5W3z8LxFrr6VbftAHI57rVLRIlSZJ2qMgF/2bNmsW1115LZvLjH/+Y0aNHs//++zNz5ky++c1v8vjjj/P444/zzW9+k5kzZ27z2kMPPZQHHnhg63yx6667rt92jj/+eL70pS8B8Mtf/pJf//rXTJpUWq/iW9/6Fo899hjd3d0sW7aM4447juOPP55ly5bxxz/+kT/84Q985Stf4fjjj2e//fbjd7/7HY8++ihPP/00X/va17a2seeee/LEE08M9T/Rjo0eO7j4ABx99NHce++9/OpXv+JPf/oT119/fZ+9p6tWreKd73wny5cv5yUvecnW+OOPP87TTz8NwCOPPMIPf/hDDj/88G1e29LSQktLC7fffjvA1u9PX44//nhuuOEGtmzZwsaNG/n+97/P9OnTgdIQ01/96lc8++yz3HDDDfzVX/0V06dP53vf+x6PPPIIW7Zs4brrruPEE08EYL/99uOee+7h2Wef5Stf+crWNqr5/bNArLXtdatLkiRpu6q14N/cuXP5y7/8S9atW8fYsWP57Gc/C8CnPvUpPvWpTwHwmte8hoMPPphDDjmEd7zjHfz7v/87AHvvvTf/9E//xNFHH83RRx/N/Pnz/2zBmz322IPFixdz+umnc9RRR21ToPT27ne/m2effZYpU6bwxje+kauvvnprb9f06dN53etex9SpU3nd615HW1sbRx11FOeddx7Tp0/nmGOO4e1vfzvTpk2jqamJ+fPnM336dE455RQOPfTQrW2cd955XHDBBbVfpGbGfGjqVcw3NZfiO2m33XbjyiuvZObMmRx22GHMmTOHl7/85QDMnz9/66qk8+bN48knn+QNb3jDNttZ3HPPPbS1tXHEEUfwqle9iosvvvjPCkSAz3/+87znPe/hyCOPJDP7zeess85i6tSpHHHEEZx88sl87GMf46UvfSlQKmYvvPBCDjvsMCZMmMBZZ53F/vvvz6WXXsqrXvUqjjjiCF7xildw5plnAnDppZfy2te+lmOPPZb9999/axvnnHMOixYtYtq0aUO+SE1s7801ira2tmxvby86jZIFLUBf/+YBC7pqnY0kSVLh7rnnHg477LABX79sVeeIXPDv6quvpr29nSuvvLLoVHbN6iWlzpFNHaWewxnzazb/sEjf/e53ueyyy7bpya2Wvv6bioiVmbnDVYlcpKbWRo8tDy/tIy5JkqQdcsG/YW7qnBFREA5XDjGttSp0q0uSJKnxnXfeecO/93AEO+mkk2rSe7irLBBrbeocOOMKGD0OiNLjGVf4VxRJkiRJhXOIaRHsVpckSdpGZu5wyX9JO7ara8zUXQ9iRJwaEesiYn1EXNzH+Q9ExNqIWB0Rt0bEQUXkKUmSpKGxxx578Oijj+7yL7bSSJeZPProo+yxxx47fY+66kGMiFHAVcApQAdwZ0Qsz8y1FZetAtoy848R8S7gY8Aba5+tJEmShsLYsWPp6Ohg48aNRaciDXt77LEHY8fu/AKYdVUgAtOB9Zl5P0BEXA+cCWwtEDPzOxXX/xh4S00zlCRJ0pBqampiwoQJRachifobYtoKVO4B0VGO9efvgFuqmpEkSZIkjRD11oM4YBHxFqANOLGf8+cD5wMceOCBNcxMkiRJkoaneutB7ATGVRyPLce2ERF/DfwvYFZmPt3XjTJzcWa2ZWbbmDFjqpKsJEmSJDWSqKfVoiJiN+CXwAxKheGdwJsy8+6Ka6YBNwKnZua9A7zvRuDBoc94l+0LPFJ0EmpYfr5UTX6+VE1+vlRtfsZUTfX6+TooM3fYc1ZXBSJARLwG+CQwCvhcZn4kIhYC7Zm5PCK+DUwBHi6/5NeZOaugdHdJRLRnZlvReagx+flSNfn5UjX5+VK1+RlTNQ33z1fdzUHMzJuBm3vF5lc8/+uaJyVJkiRJI0C9zUGUJEmSJBXEArFYi4tOQA3Nz5eqyc+XqsnPl6rNz5iqaVh/vupuDqIkSZIkqRj2IEqSJEmSAAvEQkTEqRGxLiLWR8TFReejxhMRD0TEmoi4KyLai85Hw1tEfC4ifhcRP6+I7R0R34qIe8uPexWZo4avfj5fCyKis/wz7K7yCufSoEXEuIj4TkSsjYi7I+J95bg/w7TLtvP5GtY/wxxiWmMRMYrSXo+nAB2U9nqcm5lrC01MDSUiHgDaMrMe9+DRMBMRJwBPAtdm5uRy7GPAY5l5afkPXXtl5oeKzFPDUz+frwXAk5l5WZG5afiLiP2B/TPzpxGxJ7ASmA2chz/DtIu28/mawzD+GWYPYu1NB9Zn5v2Z+SfgeuDMgnOSpH5l5veBx3qFzwSuKT+/htL/EKVB6+fzJQ2JzHw4M39afv4EcA/Qij/DNAS28/ka1iwQa68V2FBx3EEDfJBUdxL4ZkSsjIjzi05GDWm/zHy4/Pw3wH5FJqOGdGFErC4PQXX4n3ZZRIwHpgF34M8wDbFeny8Yxj/DLBClxvRXmXkUcBrwnvIQLqkqsjRXwfkKGkr/AbwMOBJ4GPh4selouIuIFwH/BfyPzPx95Tl/hmlX9fH5GtY/wywQa68TGFdxPLYck4ZMZnaWH38HfIXS0GZpKP22PPeiZw7G7wrORw0kM3+bmVsy81ng0/gzTLsgIpoo/fL+pcxcWg77M0xDoq/P13D/GWaBWHt3AhMjYkJE7A6cAywvOCc1kIh4YXmiNBHxQuDVwM+3/ypp0JYDf1N+/jfAVwvMRQ2m5xf3srPwZ5h2UkQE8Fngnsz8RMUpf4Zpl/X3+RruP8NcxbQA5aVuPwmMAj6XmR8pOCU1kIg4mFKvIcBuwJf9jGlXRMR1wEnAvsBvgf8NLAOWAAcCDwJzMtOFRjRo/Xy+TqI0NCuBB4B3VswXkwYsIv4K+AGwBni2HP6flOaJ+TNMu2Q7n6+5DOOfYRaIkiRJkiTAIaaSJEmSpDILREmSJEkSYIEoSZIkSSqzQJQkSZIkARaIkiRJkqQyC0RJkgYoIrZExF0VXxcP4b3HR8Sw2itLktR4dis6AUmShpHuzDyy6CQkSaoWexAlSdpFEfFARHwsItZExE8i4pByfHxE3BYRqyPi1og4sBzfLyK+EhE/K38dW77VqIj4dETcHRHfjIjmwt6UJGlEskCUJGngmnsNMX1jxblNmTkFuBL4ZDn2b8A1mTkV+BJwRTl+BfC9zDwCOAq4uxyfCFyVmS8HuoDXVfn9SJK0jcjMonOQJGlYiIgnM/NFfcQfAE7OzPsjogn4TWbuExGPAPtn5uZy/OHM3DciNgJjM/PpinuMB76VmRPLxx8CmjLzn6v/ziRJKrEHUZKkoZH9PB+Mpyueb8G1AiRJNWaBKEnS0HhjxeN/l5//CDin/PzNwA/Kz28F3gUQEaMiYnStkpQkaXv8y6QkSQPXHBF3VRx/IzN7trrYKyJWU+oFnFuOvRf4fETMAzYCbyvH3wcsjoi/o9RT+C7g4apnL0nSDjgHUZKkXVSeg9iWmY8UnYskSbvCIaaSJEmSJMAeREmSJElSmT2IkiSVlTe2z4hwjr4kaUSyQJQkNYyI+EZELOwjfmZE/MbCT5Kk7bNAlCQ1kmuAt0RE9IqfC3wpM58pIKchESX+f1uSVFX+j0aS1EiWAfsAx/cEImIv4LXAteXj0yNiVUT8PiI2RMSCgd48Ii6OiPsi4omIWBsRZ/U6/46IuKfi/FHl+LiIWBoRGyPi0Yi4shxfEBFfrHj9NkNcI+K7EfGRiPgh8Efg4Ih4W0Ub90fEO3vlcGZE3FV+f/dFxKkR8YaIWNnrug9ExFcH+t4lSSODBaIkqWFkZjewBHhrRXgO8IvM/Fn5+A/l8y3A6cC7ImL2AJu4j1LxORr4MPDFiNgfICLeACwo3/vFwCzg0YgYBXwNeBAYD7QC1w/ibZ0LnA/sWb7H7ygVvC+mtK/i5RWF6HRKhfC88vs7AXgAWA5MiIjDet332kHkIUkaASwQJUmN5hrg9RGxR/n4reUYAJn53cxck5nPZuZq4DrgxIHcODP/MzMfKr/2BuBeYHr59NuBj2XmnVmyPjMfLJ8/AJiXmX/IzKcy8/ZBvJ+rM/PuzHwmMzdn5tcz875yG98DvslzPaZ/B3wuM79VzrEzM3+RmU8DNwBvAYiIl1MqVr82iDwkSSOABaIkqaGUi69HgNkR8TJKBdqXe85HxDER8Z3ycM9NwAXAvgO5d0S8tTx8sysiuoDJFa8dR6mHsbdxwIO7MP9xQ68cTouIH0fEY+UcXjOAHKBUJL+pPD/zXGBJuXCUJGkrC0RJUiO6llLP4VuAFZn524pzX6Y05HJcZo4GPgX0XtTmz0TEQcCngQuBfTKzBfh5xWs3AC/r46UbgAP7WUH1D8ALKo5f2sc1WzcsjojnA/8FXAbsV87h5gHkQGb+GPgTpd7GNwFf6Os6SdLIZoEoSWpE1wJ/DbyDiuGlZXsCj2XmU+U5e28a4D1fSKlY2wgQEW+j1IPY4zPAP0TEK8orjh5SLip/AjwMXBoRL4yIPSLiuPJr7gJOiIgDI2I0cMkOctgdeH45h2ci4jTg1RXnPwu8LSJmRMTzIqI1Ig6tOH8tcCWweZDDXCVJI4QFoiSp4WTmA8CPKBV1y3udfjewMCKeAOZTWtRmIPdcC3wc+G/gt8AU4IcV5/8T+AilHsonKK2oundmbgHOAA4Bfg10AG8sv+ZblOYGrgZWsoM5gZn5BHBROefHKRW3yyvO/4TywjXAJuB7wEEVt/gCpaL2i0iS1IfIzB1fJUmShr2IaKa0CupRmXlv0flIkuqPPYiSJI0c7wLutDiUJPWnrwnzkiSpwUTEA5QWsxnono+SpBHIIaaSJEmSJMAhppIkSZKkMgtESZIkSRIwQuYg7rvvvjl+/Pii05AkSZKkQqxcufKRzByzo+tGRIE4fvx42tvbi05DkiRJkgoREQ8O5DqHmEqSJEmSAAtESZIkSVKZBaIkSZIkCbBAlCRJkiSVWSBKkiRJkgALREmSJElSmQWiJEmqD6uXwOWTYUFL6XH1kqIzkqQRp6oFYkScGhHrImJ9RFzcx/kLImJNRNwVEbdHxOHl+PiI6C7H74qIT1W85hXl16yPiCsiIqr5HiRJUg2sXgI3XQSbNgBZerzpIotESaqxqhWIETEKuAo4DTgcmNtTAFb4cmZOycwjgY8Bn6g4d19mHln+uqAi/h/AO4CJ5a9Tq/UeJElSjdy6EDZ3bxvb3F2KS5Jqppo9iNOB9Zl5f2b+CbgeOLPygsz8fcXhC4Hc3g0jYn/gxZn548xM4Fpg9tCmLUmSam5Tx+DikqSqqGaB2ApsqDjuKMe2ERHviYj7KPUgXlRxakJErIqI70XE8RX3rPw/RZ/3lCRJw8zosYOLS5KqovBFajLzqsx8GfAh4B/L4YeBAzNzGvAB4MsR8eLB3Dcizo+I9oho37hx49AmLUmShtaM+dDUvG2sqbkUlyTVTDULxE5gXMXx2HKsP9dTHi6amU9n5qPl5yuB+4C/KL++8k+J/d4zMxdnZltmto0ZM2an34QkSaqBqXPgjCtg9DggSo9nXFGKS5JqZrcq3vtOYGJETKBUxJ0DvKnygoiYmJn3lg9PB+4tx8cAj2Xmlog4mNJiNPdn5mMR8fuIeCVwB/BW4N+q+B4kSVKtTJ1jQShJBatagZiZz0TEhcAKYBTwucy8OyIWAu2ZuRy4MCL+GtgMPA78TfnlJwALI2Iz8CxwQWY+Vj73buBqoBm4pfwlSZIkSdpFUVoMtLG1tbVle3t70WlIkiRJUiEiYmVmtu3ousIXqZEkSZIk1QcLREmSJEkSYIEoSZIkSSqzQJQkSZIkARaIkiSpt9VL4PLJsKCl9Lh6SdEZSZJqpJr7IEqSpOFm9RK46SLY3F063rShdAzuUShJI4A9iJIk6Tm3LnyuOOyxubsUlyQ1PAtESZL0nE0dg4tLkhqKBaIkSXrO6LGDi0uSGooFoiRJes6M+dDUvG2sqfn/t3f/cVbXZeL3X1cDyKh9GTUyHVAx+eIaILQDdWv+ihRLRTKXtKzsl1qZ23cfS+K95aL3dsuKmz1I793cSm23VHKVaFvD8kct/TAGYVFREn8UM7orqFjWoIjX/cc5g2doBgZmznyYc17Px+M8zvlc5/P5nPc5czic67zf7+tdikuSap4JoiRJes3EWXDaAhgxGojS9WkLLFAjSXXCKqaSJKmribNMCCWpTtmDKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJGjirFsLV42FuU+l61cKiWySpQlUTxIg4OSLWRMTaiJjTzf0XRMQDEbEyIpZGxBHl+IkRsbx83/KIeGfFMfeWz7myfHljNZ+DJEmS+smqhfD9i+CFdUCWrr9/kUmitBupWoIYEQ3AtcC7gSOAszsTwArfycwJmTkJuBL4cjm+ATgtMycAHwH+ZZvjPpiZk8qXZ6r1HCRJktSP7rocNnd0jW3uKMUl7Raq2YM4FVibmY9n5svAzcDplTtk5u8qNvcCshxfkZlPleMPAY0RsUcV2ypJkqRqe6Ft5+KSBlw1E8RmYF3Fdls51kVEfCYiHqPUg3hRN+d5H3B/Zr5UEbu+PLz0ixER/dloSZIkVcmIUTsXlwaTGplfW3iRmsy8NjPfDFwMfKHyvoh4C/D3wPkV4Q+Wh54eU758qLvzRsR5EdEaEa3r16+vTuMlSZLUe9MuhaGNXWNDG0txaTCrofm11UwQ24HRFdujyrGe3AzM7NyIiFHA7cCHM/Oxznhmtpevfw98h9JQ1j+RmddlZktmtowcOXKXn4QkSZL6ycRZcNoCGDEaiNL1aQtKcWkwq6H5tUOqeO5lwNiIGEMpMTwL+EDlDhExNjMfLW+eAjxajjcBPwDmZObPKvYfAjRl5oaIGAqcCvy4is9BkiRJ/WniLBNC1Z4aml9btQQxM1+JiAuBJUAD8M3MfCgiLgdaM3MxcGFEvAvYDDxPqWIpwIXAYcClEdE55uAk4A/AknJy2EApOfznaj0HSZIkSdqhEaPKw0u7iQ8ykZlFt6HqWlpasrW1tehmSJIk7TYWrWhn/pI1PLWxgwObGpk9fRwzJ/9JPUFJvdE5B7FymOnQxt1qCHVELM/Mlh3tV80hppIkSdoNLVrRziW3PUDH5i0AtG/s4JLbHgAwSayGVQtLc9FeaCv1KE27dLdJGtRPOv+eNfB3NkGUJEmqM/OXrNmaHHbq2LyF+UvWmCD2t217ljqrW0LVkwd7iQdYjcyvLXyZC0mSJA2spzZ27FRcfVBQdcvOXuL2jR0kr/USL1qxvUUFJBNESZKkunNgUyMzXreUpcMu4vE9PsDSYRcx43VLObCpcccHa+cUVN1ye73E0vY4xFSSJKkgRQ0B/MoRjzJ++ddpjJcBGBUb+PuhX+fBIw4B3ln1xy9KIa93QdUt67aX2PmefWYPoiRJUgGKHAI45bGvbk0OOzXGy0x57KtVf+yiFPV6L3vzZ+nIYV1iHTmMZW/+bFUft6fe4JruJe6c7/nCOiBfm++5amHRLRtUTBAlSZIKUOgQwBpa1Lu3inq9P7d6LBdv/gRtr76BVzNoe/UNXLz5E3xu9diqPu7s6eNoHNrQJdY4tIHZ08dV9XELVdB8z1rjEFNJkqQCFDoEsIYW9e6tol7vpzZ20M47WPzyO7rEo8qP2zl0tq6qmNbhDx/VYIIoSZJUgAObGmnvJkkYkCGA0y7tflHvaZdW/aGLmndZ1Otd5N955uTm4hLCIuYC1uEPH9XgEFNJkqQCFDoEcOIsOG0BjBgNROn6tAUDsi5fUfMuZ08fx5nDft6lcuuZw35e9de7Lod6FjUXcNqlpR86Kg3QDx+1JDKz6DZUXUtLS7a2thbdDEm1xkppkvqo3hYyP3re3d32pjU3NfKzOVWunrpqIa9877MM2bJpa+iVhuEMOf2rLljf364e30NP3mj4Pw9W97H9v7lHEbE8M1t2uJ8JoiTtgs5fR7cdnjUAv8BL1VZ3X2Y1YMbM+QHdffMM4Il5p1T3wYtMWurN3Cbo6S89d+NAt0ZlvU0QHWIqSbvCSmmqUUUOAVTtK3TpBQuYDJye5vw5F3BQMEGUpF3hFw3VqEKXXlDNK3Q+nknLwJl2Ka80DO8SeqVhuHMBBwkTREnaFX7R0ABYtKKdo+fdzZg5P+DoeXcPSFPfyWAAACAASURBVC9eoUsvqObNnNzMFWdMoLmpkaA09/CKMyYMzBBmC5gMmEVbjmbONms/ztn8CRZtObropqkXXOZCknZFgSXiVR86h3p29uZ1DvUEqvplutClF1QXClt6oXN+uAVMqm7+kjW0v3wUt3JUl/gvlqxxPvMgYIIoSbvCLxoDr84q021vqGc1v2DNnj6uS2IKdVCSX/Vj4qya/tzYXTgSYXAzQZSkXeUXjYGzbdXYzjW1oGb/BkV9wepMPq1iKmlXORJhcDNBlCTt/rZXNbZGE8Qiv2AVNgRQUk1wJMLgZoIoSdr91WHV2NnTx7H09v+Pz3EzB8YGnso38BXO4h3TP1100yRpu+p1JEKtrCFrgihJ2v2NGNXDAtfVrxpb1H/4Mxt+xqlDv86QLZsAGBUbmNfwdYY0HAnUZq9pkZYt/hqj75/PG3M9z8RI1r11NlNmnF90s6RBq95GIhRVWKwaXOZCkrT7K6g8faGLxt91+dbksNOQLZtKw2rVr5Yt/hrjl3+BN7Ge1wW8ifWMX/4Fli3+WtFNk7STilgeCGprDdmqJogRcXJErImItRExp5v7L4iIByJiZUQsjYgjKu67pHzcmoiY3ttzSpJq0MRZcNoCGDEaiNL1aQuqPv+w0P/w63BYbVFG3z+fxni5S6wxXmb0/fMLapGkXVHkj3q1VLm1akNMI6IBuBY4EWgDlkXE4sxcXbHbdzLzn8r7zwC+DJxcThTPAt4CHAj8OCL+d/mYHZ1TklQtRS41UUDV2EL/wy9wWG29eWOuh+guvmHgGyP1J5cHGpDlgaC2KrdWswdxKrA2Mx/PzJeBm4HTK3fIzN9VbO4FZPn26cDNmflSZj4BrC2fb4fnlOpdUUMrVAc6l5p4YR2Qry01sWph0S2rmp7+Yx+Q//ALGlZbj56JkT3E3zDALZH6UR1+Zhf5o97s6eNoHNrQJTZYK7dWM0FsBip/+mwrx7qIiM9ExGPAlcBFOzi2V+eUdgurFsLV42FuU+l6AD6QC50vVaQCXuu6tL2lJmpUof/hFzSsth6te+tsOnJYl1hHDmPdW2cX1CKpH9ThZ3aRP+rNnNzMFWdMoLmpkQCamxq54owJg65ADewGVUwz81rg2oj4APAF4CP9cd6IOA84D+Cggw7qj1NKvVfQot7zl6zhxC0/4fPDFm4ti3/lK7OYv2TYoPyA6pU6XEC9MHU4J67wUu0FDKutR1NmnM8yKFcx3cAz8QbW/blVTDXI1eFndtHrL9ZK5dZqJojtwOiK7VHlWE9uBv6xF8f26pyZeR1wHUBLS0t2t49UNQUt6t3yux9xxdCvs2e52MKo2MC8oV/nkt8BvLNqj1uoOlxAvTB1OieuVv7D1/ZNmXE+lBPCN5Uv0qBWh5/Zhf+oVyOqmSAuA8ZGxBhKSdxZwAcqd4iIsZn5aHnzFKDz9mLgOxHxZUpFasYCv6I0hXy755R2CwX9anfJsO+yJ10r8e0ZL3PJsO8CV1T1sQtTh7+QFmbapV17a8E5cZK0u6rTz2x/1Ou7qiWImflKRFwILAEagG9m5kMRcTnQmpmLgQsj4l3AZuB5ysNLy/stBFYDrwCfycwtAN2ds1rPQdplBf1qtz/dV9zrKV4T6vAX0sJ09sjWUUU8SRq0/MzWLorM2h992dLSkq2trUU3Q/Vk23lxUPrVrtoFJq4e30OyNBr+z4PVe9wiFfVaS5IkDSIRsTwzW3a0XzWrmEr1q6jqg/VYFt9Kj5IkSf3GHkSp1tTZoriSJEnasd72IBa+zIWkfmZZfEmSJO0iE0RJkrRbWLSivf7K0zvqQ9JuxgRRUv/wS46kPli0or3LAtftGzu45LYHAGo3Sdy2yNYL60rb4OenpMJYpEZS33V+yXlhHZCvfclZtbDoltWuVQtLVWvnNpWufa01yM1fsmZrctipY/MW5i9ZU1CLBsBdl3etwAyl7bsuL6Y9koQJoqT+4JecgWVCrhr01MaOnYrXhBfadi4uSQPABFFS3/klZ2CZkKsGHdjUuFPxmjBi1M7FJWkAmCBK6ju/5AwsE3LVoNnTx9E4tKFLrHFoA7OnjyuoRQOgHteulbTbM0GU1Hd+yRlYJuSqQTMnN3PFGRNobmokgOamRq44Y0LtFqiBUiGa0xbAiNFAlK5PW2CBGkmFiswsug1V19LSkq2trUU3Q6ptVjEdONtWPoRSQu4XS0mS1IOIWJ6ZLTvaz2UuJPWPibNMTgZK5+tsQi5JkvqZCaIkDUYm5JIkqQqcgyhJkiRJAuxBlNRPFq1oZ/6SNTy1sYMDmxqZPX1cbReXkCRJqkEmiJL6bNGKdi657QE6Nm8BoH1jB5fc9gCASaIkSdIg4hBTSX02f8marclhp47NW5i/ZE1BLZIkSdKuMEGU1GdPbezYqbg0qKxaCFePh7lNpetVC4tukSRJVbPDBDEiPhsR+wxEYyQNTgc2Ne5UXBo0OtecfGEdkKXr719kkihJqlm96UHcH1gWEQsj4uSIiGo3StLgMnv6OBqHNnSJNQ5tYPb0cQW1SOond10Om7fpCd/cUYpLklSDdlikJjO/EBFfBE4CPgpcExELgW9k5mPVbqDUF1bWHBidr6mvtaqpkH/PL7TtXFySpEGuV1VMMzMj4r+B/wZeAfYBbo2IH2Xm56vZQGlXWVlzYM2c3Ozrqqop7N/ziFHl4aXdxCVJqkG9mYP4lxGxHLgS+BkwITM/Bfw58L4dHHtyRKyJiLURMaeb+/8qIlZHxKqIuCsiDi7HT4iIlRWXTRExs3zfDRHxRMV9k3bheWuALVrRztHz7mbMnB9w9Ly7WbSiveqPOX/JGk7c8hOWDruIx/f4AEuHXcSJW35iZU1pECqsUu60S2HoNnNphzaW4pIk1aDe9CDuC5yRmb+pDGbmqxFxak8HRUQDcC1wItBGaR7j4sxcXbHbCqAlM/8YEZ+ilIS+PzPvASaVz7MvsBa4s+K42Zl5ay/art1AUb/8t/zuR1wx9OvsGS8DMCo2MG/o17nkdwDvrNrjSgOh3oZPF1Ypd+Ks0vVdl5eGlY4YVUoOO+OSJNWY3iSIdwDPdW5ExP8C/iwz78vMh7dz3FRgbWY+Xj7uZuB0YGuCWE4EO/0SOKeb85wJ3JGZf+xFW7Ub2t4v/9X8QnvJsO+yJy93ie0ZL3PJsO8CV1TtcaVqq8fh0wc2NdLeTTI4IJVyJ84yIZQk1Y3eVDH9R+DFiu0Xy7EdaQYqJ260lWM9+TilZHRbZwE3bRP7UnlY6tURsUcv2qICFfXL//5s2Km4NFgUNtyyQFbKlSRpYPQmQYzMzM6NzHyVXha36a2IOAdoAeZvEz8AmAAsqQhfAhwOTKE0/PXiHs55XkS0RkTr+vXr+7O52klFrZEXPRSR6CkuDRaFDbcs0MzJzVxxxgSamxoJoLmpkSvOmFCzPaaSJBWlN4ne4xFxEa/1Gn4aeLwXx7UDoyu2R5VjXUTEu4C/AY7LzJe2uXsWcHtmbu4MZObT5ZsvRcT1wF939+CZeR1wHUBLS0t2t48Gxuzp47oMh4MB+uV/2qWlBa0r1zCzuIRqQKHDLQtkpVxJkqqvNz2IFwBHUUru2oC3Aef14rhlwNiIGBMRwygNFV1cuUNETAa+BszIzGe6OcfZbDO8tNyrSEQEMBN4sBdtUYEK++V/4iw4bQGMGA1E6fq0BQM2l6iIyq2qDw63lCRJ1RIVo0f7/+QR7wG+AjQA38zML0XE5UBrZi6OiB9TGkLa2Sv428ycUT72EErLaowuD2vtPOfdwEgggJXABZlZOUfyT7S0tGRra2u/Pjdpe7YtIgKlL/AOiVN/qbcqppIkqW8iYnlmtuxwvx0liBExnFIBmbcAwzvjmfmxvjZyoJggaqAdPe/ubocANjc18rM5LrFRK0zSJEnSYNHbBLE3Q0z/BXgTMB34CaW5hL/vW/Ok2laPRUTqTWcvcfvGDpLXlppwKLEkSRrMepMgHpaZXwT+kJk3AqdQmocoqQdFVW7VwKnHpSYkSVLt602C2FlBdGNEjAdGAG+sXpOkwc8iIrXPXmJJklSLerPMxXURsQ/wBUpVSPcGvljVVkmDXOc8NOen1a56XWpCkiTVtu0miBHxOuB3mfk88FPg0AFplVQDXLOtthW2vqckSVIVbXeIaXl5ic8PUFskadAobH1PSZKkKurNENMfR8RfA7cAf+gMZuZzVWuVJA0C9hJLkqRa05sE8f3l689UxBKHm0qSJElSTdlhgpiZYwaiIZIkSZKkYu0wQYyID3cXz8xv9X9zJEm7s0Ur2q3OK0lSDevNENMpFbeHA9OA+wETREmqI4tWtHep3Nq+sYNLbnsAwCRRkqQa0Zshpp+t3I6IJuDmqrVIkrRbmr9kTZdlPQA6Nm9h/pI1JoiSJNWI7S5z0YM/AM5LlKQ689TGjp2KS5Kkwac3cxC/T6lqKZQSyiOAhdVslKrDuUOS+uLApkbau0kGD2xqLKA1kiSpGnozB/GqituvAL/JzLYqtUdV4twhSX01e/q4Lp8jAI1DG5g9fVyBrZIkSf2pNwnib4GnM3MTQEQ0RsQhmflkVVumfuXcIUl91flZ4UgESZJqV28SxO8CR1VsbynHpnS/u3ZHzh2S1B9mTm42IZQkqYb1JkEckpkvd25k5ssRMayKbVIVOHdItcz5tZIkSf2jN1VM10fEjM6NiDgd2FC9JqkaZk8fR+PQhi6xAZ07tGohXD0e5jaVrldZ50j9o3N+bfvGDpLX5tcuWtFedNMkSZIGnd70IF4AfDsirilvtwEfrl6TVA2Fzh1atRC+fxFsLvdgvrCutA0wcVb1H181zfm1kiRJ/WeHCWJmPga8PSL2Lm+/WPVWqSoKmzt01+WvJYedNneU4iaI6iPn10qSJPWfHQ4xjYj/NyKaMvPFzHwxIvaJiL8biMapRrzQw6ooPcWlndDTPFrn10qSJO283sxBfHdmbuzcyMzngff05uQRcXJErImItRExp5v7/yoiVkfEqoi4KyIOrrhvS0SsLF8WV8THRMR95XPeYsGcQWDEqJ2LSzuh8Pm1kiRJNaQ3CWJDROzRuRERjcAe29m/c78G4Frg3cARwNkRccQ2u60AWjJzInArcGXFfR2ZOal8mVER/3vg6sw8DHge+HgvnoOKNO1SGLpNb87QxlJc6qOZk5u54owJNDc1EkBzUyNXnDHB+YeSJEm7oDdFar4N3BUR1wMBnAvc2IvjpgJrM/NxgIi4GTgdWN25Q2beU7H/L4FztnfCiAjgncAHyqEbgbnAP/aiPSpK5zzDuy4vDSsdMaqUHDr/UP3EtfkkSZL6R2+K1Px9RPwX8C4ggSXAwds/CoBmYF3Fdhvwtu3s/3Hgjort4RHRCrwCzMvMRcB+wMbMfKXinH4rHAwmzjIhlCRJknZzvelBBPgfSsnhXwBPAP/Wn42IiHOAFuC4ivDBmdkeEYcCd0fEA8ALO3HO84DzAA466KD+bK4kSZIk1aQe5yBGxP+OiL+NiEeArwK/BSIzT8jMa3o6rkI7MLpie1Q5tu3jvAv4G2BGZr7UGc/M9vL148C9wGTgWaApIjoT227PWT7uusxsycyWkSNH9qK5kiRJklTftlek5hFK8/1Ozcx3ZOZXgS3b2X9by4Cx5aqjw4CzgMWVO0TEZOBrlJLDZyri+3QWxomINwBHA6szM4F7gDPLu34E+N5OtEmSJEmS1IPtJYhnAE8D90TEP0fENEpFanqlPE/wQkpzFh8GFmbmQxFxeUR0ViWdD+wNfHeb5Sz+DGgtz328h9IcxM7iNhcDfxURaynNSfxGb9skSZIkSepZlDrltrNDxF6Uqo+eTalH8VvA7Zl5Z/Wb1z9aWlqytbW16GZIkiRJUiEiYnlmtuxovx2ug5iZf8jM72TmaZTm/K2g1IsnSZIkSaohO0wQK2Xm8+XiL9Oq1SBV0aqFcPV4mNtUul61sOgWSZIkSdqN9HaZCw12qxbC9y+CzR2l7RfWlbbB9QklSZIkATvZg6hB7K7LX0sOO23uKMUlSZIkCRPE+vFC287FJUmSJNUdE8R6MWLUzsUlSZIk1R0TxHox7VIY2tg1NrSxFJckSZIkTBDrx8RZcNoCGDEaiNL1aQssUCNJkiRpK6uY1pOJs0wIJUmSJPXIHkRJkiRJEmCCKEmSJEkqM0GUJEmSJAEmiJIkSZKkMhNESZIkSRJggihJkiRJKjNBlCRJkiQBJoiSJEmSpDITREmSJEkSYIIoSZIkSSozQZQkSZIkASaIkiRJkqQyE0RJkiRJElDlBDEiTo6INRGxNiLmdHP/X0XE6ohYFRF3RcTB5fikiPhFRDxUvu/9FcfcEBFPRMTK8mVSNZ+DJEmSJNWLqiWIEdEAXAu8GzgCODsijthmtxVAS2ZOBG4FrizH/wh8ODPfApwMfCUimiqOm52Zk8qXldV6DpIkSZJUT6rZgzgVWJuZj2fmy8DNwOmVO2TmPZn5x/LmL4FR5fivM/PR8u2ngGeAkVVsqyRJkiTVvWomiM3AuorttnKsJx8H7tg2GBFTgWHAYxXhL5WHnl4dEXv0R2MlSZIkqd7tFkVqIuIcoAWYv038AOBfgI9m5qvl8CXA4cAUYF/g4h7OeV5EtEZE6/r166vWdkmSJEmqFdVMENuB0RXbo8qxLiLiXcDfADMy86WK+P8CfgD8TWb+sjOemU9nyUvA9ZSGsv6JzLwuM1sys2XkSEenSpIkSdKOVDNBXAaMjYgxETEMOAtYXLlDREwGvkYpOXymIj4MuB34Vmbeus0xB5SvA5gJPFjF5yBJkiRJdWNItU6cma9ExIXAEqAB+GZmPhQRlwOtmbmY0pDSvYHvlvI9fpuZM4BZwLHAfhFxbvmU55Yrln47IkYCAawELqjWc5AkSZKkehKZWXQbqq6lpSVbW1uLboYkSZIkFSIilmdmy4722y2K1EiSJEmSimeCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQKqnCBGxMkRsSYi1kbEnG7u/6uIWB0RqyLirog4uOK+j0TEo+XLRyrifx4RD5TPuSAioprPQZIkSZLqRdUSxIhoAK4F3g0cAZwdEUdss9sKoCUzJwK3AleWj90X+FvgbcBU4G8jYp/yMf8IfBIYW76cXK3nIEmSJEn1pJo9iFOBtZn5eGa+DNwMnF65Q2bek5l/LG/+EhhVvj0d+FFmPpeZzwM/Ak6OiAOA/5WZv8zMBL4FzKzic5AkSZKkulHNBLEZWFex3VaO9eTjwB07OLa5fLu355QkSZIk9dKQohsAEBHnAC3Acf14zvOA8wAOOuig/jqtJEmSJNWsavYgtgOjK7ZHlWNdRMS7gL8BZmTmSzs4tp3XhqH2eE6AzLwuM1sys2XkyJG7/CQkSZIkqV5UM0FcBoyNiDERMQw4C1hcuUNETAa+Rik5fKbiriXASRGxT7k4zUnAksx8GvhdRLy9XL30w8D3qvgcJEmSJKluVG2IaWa+EhEXUkr2GoBvZuZDEXE50JqZi4H5wN7Ad8urVfw2M2dk5nMR8f9QSjIBLs/M58q3Pw3cADRSmrN4B5IkSZKkPotSMdDa1tLSkq2trUU3Q5IkSZIKERHLM7NlR/tVc4ipJEmSJGkQMUGUJEmSJAEmiJIkSZKkMhNESZIkSRJggihJkiRJKjNBlCRJkiQBJoiSJEmSpDITREmSJEkSYIIoSZIkSSozQZQkSZIkASaIkiRJkqQyE0RJkiRJEgBDim5APVq0op35S9bw1MYODmxqZPb0ccyc3Fx0syRJkgqxefNm2tra2LRpU9FNkQa94cOHM2rUKIYOHbpLx5sgDrBFK9q55LYH6Ni8BYD2jR1cctsDACaJkiSpLrW1tfH617+eQw45hIgoujnSoJWZPPvss7S1tTFmzJhdOodDTAfY/CVrtiaHnTo2b2H+kjUFtUiSJKlYmzZtYr/99jM5lPooIthvv/361BtvgjjAntrYsVNxSZKkemByKPWPvv5bMkEcYAc2Ne5UXJIkSdX3sY99jDe+8Y2MHz++x30yk4suuojDDjuMiRMncv/992+978Ybb2Ts2LGMHTuWG2+8cYePd++993Lqqaf2S9t3xQ033MBTTz1V2OP3tx/+8IeMGzeOww47jHnz5nW7z5e//GWOOOIIJk6cyLRp0/jNb36z9b6GhgYmTZrEpEmTmDFjxg4f78knn9zue6XaFi1axOrVq6tybhPEATZ7+jgahzZ0iTUObWD29HEFtUiSJGlwWbSinaPn3c2YOT/g6Hl3s2hFe5/Pee655/LDH/5wu/vccccdPProozz66KNcd911fOpTnwLgueee47LLLuO+++7jV7/6FZdddhnPP//8LrXjlVde2aXjdlahCeKqhXD1eJjbVLpetbBPp9uyZQuf+cxnuOOOO1i9ejU33XRTt8nT5MmTaW1tZdWqVZx55pl8/vOf33pfY2MjK1euZOXKlSxevHiX2zJQfz8TxBoyc3IzV5wxgeamRgJobmrkijMmWKBGkiSpFzoL/rVv7CB5reBfX5PEY489ln333Xe7+3zve9/jwx/+MBHB29/+djZu3MjTTz/NkiVLOPHEE9l3333ZZ599OPHEE7tNNn/4wx9y+OGH89a3vpXbbrtta3zu3Ll86EMf4uijj+ZDH/oQmzZt4qMf/SgTJkxg8uTJ3HPPPUApqTv99NM5/vjjGTt2LJdddtnWc3z5y19m/PjxjB8/nq985SvAn/ZyXXXVVcydO5dbb72V1tZWPvjBDzJp0iQ6OgZwqtOqhfD9i+CFdUCWrr9/UZ+SxF/96lccdthhHHrooQwbNoyzzjqL733ve3+y3wknnMCee+4JwNvf/nba2tp26nGWL1/OkUceyZFHHsm11167NX7DDTcwY8YM3vnOdzJt2jQyk9mzZzN+/HgmTJjALbfcApR6jY899lhOOeUUxo0bxwUXXMCrr74KwE033cSECRMYP348F1988dZz77333ltv33rrrZx77rn8/Oc/Z/HixcyePZtJkybx2GOP7dTz2BGrmBZg5uRmE0JJkqRdsL2Cf9X+ftXe3s7o0aO3bo8aNYr29vYe45U2bdrEJz/5Se6++24OO+ww3v/+93e5f/Xq1SxdupTGxkb+4R/+gYjggQce4JFHHuGkk07i17/+NVBKhh588EH23HNPpkyZwimnnEJEcP3113PfffeRmbztbW/juOOOY5999un2eZx55plcc801XHXVVbS0tPTXy9M7d10Om7dJSDd3lOITZ+3SKbt7/e+7777tHvONb3yDd7/73Vu3N23aREtLC0OGDGHOnDnMnDnzT4756Ec/yjXXXMOxxx7L7Nmzu9x3//33s2rVKvbdd1/+7d/+jZUrV/Jf//VfbNiwgSlTpnDssccCpb/f6tWrOfjggzn55JO57bbbOOqoo7j44otZvnw5++yzDyeddBKLFi3qtg0ARx11FDNmzODUU0/lzDPP7PXr1Fv2IEqSJGnQGKwF/x555BHGjBnD2LFjiQjOOeecLvfPmDGDxsZSTYqlS5duvf/www/n4IMP3pognnjiiey33340NjZyxhlnsHTpUpYuXcp73/te9tprL/bee2/OOOMM/vM//3Ngn2BvvdBDr11P8Sr413/9V1pbW7skeb/5zW9obW3lO9/5Dp/73Of+pFdu48aNbNy4cWui96EPfajL/Z09yFD6+5199tk0NDSw//77c9xxx7Fs2TIApk6dyqGHHkpDQwNnn302S5cuZdmyZRx//PGMHDmSIUOG8MEPfpCf/vSn1XwJtssEsQj9PO5akiSpXhRZ8K+5uZl169Zt3W5ra6O5ubnH+M7Ya6+9erXfthUqt1excsiQIVuHMAJ9Wvqg34wYtXPxXtiZ1//HP/4xX/rSl1i8eDF77LFHl3MAHHrooRx//PGsWLFip9pQjb/ftvcP1N+vqgliRJwcEWsiYm1EzOnm/mMj4v6IeCUizqyInxARKysumyJiZvm+GyLiiYr7JlXzOfS7Koy7liRJqhdFFvybMWMG3/rWt8hMfvnLXzJixAgOOOAApk+fzp133snzzz/P888/z5133sn06dO7HHv44Yfz5JNPbu2Zuummm3p8nGOOOYZvf/vbAPz617/mt7/9LePGlZ7fj370I5577jk6OjpYtGgRRx99NMcccwyLFi3ij3/8I3/4wx+4/fbbOeaYY9h///155plnePbZZ3nppZf493//962P8frXv57f//73/f0S7di0S2HoNsn80MZSfBdNmTKFRx99lCeeeIKXX36Zm2++udtKpCtWrOD8889n8eLFvPGNb9waf/7553nppZcA2LBhAz/72c844ogjuhzb1NREU1MTS5cuBdj69+nOMcccwy233MKWLVtYv349P/3pT5k6dSpQGmL6xBNP8Oqrr3LLLbfwjne8g6lTp/KTn/yEDRs2sGXLFm666SaOO+44APbff38efvhhXn31VW6//fatj1HNv1/V5iBGRANwLXAi0AYsi4jFmVlZbue3wLnAX1cem5n3AJPK59kXWAvcWbHL7My8tVptr6oqjLuWJEmqF53zDOcvWcNTGzs4sKmR2dPH9Xn+4dlnn829997Lhg0bGDVqFJdddhkf//jH+ad/+icALrjgAt7znvfwH//xHxx22GHsueeeXH/99QDsu+++fPGLX2TKlCkAXHrppX9S8Gb48OFcd911nHLKKey5554cc8wxPX7B//SnP82nPvUpJkyYwJAhQ7jhhhu29nZNnTqV973vfbS1YDyWyAAACGRJREFUtXHOOedsnUN47rnnbk1CPvGJTzB58uStbZk6dSrNzc0cfvjhWx/j3HPP5YILLqCxsZFf/OIXW4e3Vl3n9927Li8NKx0xqpQc9uF78JAhQ7jmmmuYPn06W7Zs4WMf+xhvectbgNLzb2lpYcaMGcyePZsXX3yRv/iLvwDgoIMOYvHixTz88MOcf/75vO51r+PVV19lzpw5f5IgAlx//fV87GMfIyI46aSTemzPe9/7Xn7xi19w5JFHEhFceeWVvOlNb+KRRx5hypQpXHjhhaxdu5YTTjiB9773vbzuda9j3rx5nHDCCWQmp5xyCqeffjoA8+bN49RTT2XkyJG0tLTw4osvAnDWWWfxyU9+kgULFnDrrbfy5je/eZdfv21FZvbbybqcOOL/AuZm5vTy9iUAmXlFN/veAPx7d0lfRJwHHJeZH9zRvj1paWnJ1tbWXXka/W9uE9Ddax4wd+NAt0aSJKlwDz/8MH/2Z39WdDN2ezfccAOtra1cc801RTdFu+Dee+/lqquu6tKTWy3d/ZuKiOWZucOqRNUcYtoMrKvYbivHdtZZwLZ98F+KiFURcXVE7NHdQbutKoy7liRJkqT+sFsXqYmIA4AJwJKK8CXA4cAUYF/g4m4OJSLOi4jWiGhdv3591dvaa1UYdy1JkqTad+6559p7OIgdf/zxA9J72FfVTBDbgdEV26PKsZ0xC7g9Mzd3BjLz6Sx5CbgemNrdgZl5XWa2ZGbLyJEjd/Jhq2jiLDhtAYwYDUTp+rQFzj+UJEmSVLiqFakBlgFjI2IMpcTwLOADO3mOsyn1GG4VEQdk5tNRqvk6E3iwPxo7oCbOMiGUJEmqkJk7LPkvacf6WmOmaj2ImfkKcCGl4aEPAwsz86GIuDwiZgBExJSIaAP+AvhaRDzUeXxEHEKpB/In25z62xHxAPAA8Abg76r1HCRJklR9w4cP59lnn+3zF1up3mUmzz77LMOHD9/lc1StiunuZLeqYipJkqQuNm/eTFtb2+6xkLs0yA0fPpxRo0YxdOjQLvHeVjGt5hBTSZIkaYeGDh3KmDFjim6GJHbzKqaSJEmSpIFjgihJkiRJAkwQJUmSJElldVGkJiLWA78puh3deAOwoehGqGb5/lI1+f5SNfn+UrX5HlM17a7vr4Mzc4cLxNdFgri7iojW3lQSknaF7y9Vk+8vVZPvL1Wb7zFV02B/fznEVJIkSZIEmCBKkiRJkspMEIt1XdENUE3z/aVq8v2lavL9pWrzPaZqGtTvL+cgSpIkSZIAexAlSZIkSWUmiAWIiJMjYk1ErI2IOUW3R7UnIp6MiAciYmVEtBbdHg1uEfHNiHgmIh6siO0bET+KiEfL1/sU2UYNXj28v+ZGRHv5M2xlRLynyDZq8IqI0RFxT0SsjoiHIuIvy3E/w9Rn23l/DerPMIeYDrCIaAB+DZwItAHLgLMzc3WhDVNNiYgngZbM3B3X4NEgExHHAi8C38rM8eXYlcBzmTmv/EPXPpl5cZHt1ODUw/trLvBiZl5VZNs0+EXEAcABmXl/RLweWA7MBM7FzzD10XbeX7MYxJ9h9iAOvKnA2sx8PDNfBm4GTi+4TZLUo8z8KfDcNuHTgRvLt2+k9B+itNN6eH9J/SIzn87M+8u3fw88DDTjZ5j6wXbeX4OaCeLAawbWVWy3UQNvJO12ErgzIpZHxHlFN0Y1af/MfLp8+7+B/YtsjGrShRGxqjwE1eF/6rOIOASYDNyHn2HqZ9u8v2AQf4aZIEq16R2Z+Vbg3cBnykO4pKrI0lwF5yuoP/0j8GZgEvA08A/FNkeDXUTsDfwb8LnM/F3lfX6Gqa+6eX8N6s8wE8SB1w6MrtgeVY5J/SYz28vXzwC3UxraLPWn/ynPveicg/FMwe1RDcnM/8nMLZn5KvDP+BmmPoiIoZS+vH87M28rh/0MU7/o7v012D/DTBAH3jJgbESMiYhhwFnA4oLbpBoSEXuVJ0oTEXsBJwEPbv8oaactBj5Svv0R4HsFtkU1pvOLe9l78TNMuygiAvgG8HBmfrniLj/D1Gc9vb8G+2eYVUwLUC51+xWgAfhmZn6p4CaphkTEoZR6DQGGAN/xPaa+iIibgOOBNwD/A/wtsAhYCBwE/AaYlZkWGtFO6+H9dTyloVkJPAmcXzFfTOq1iHgH8J/AA8Cr5fD/TWmemJ9h6pPtvL/OZhB/hpkgSpIkSZIAh5hKkiRJkspMECVJkiRJgAmiJEmSJKnMBFGSJEmSBJggSpIkSZLKTBAlSeqliNgSESsrLnP68dyHRMSgWitLklR7hhTdAEmSBpGOzJxUdCMkSaoWexAlSeqjiHgyIq6MiAci4lcRcVg5fkhE3B0RqyLirog4qBzfPyJuj4j/Kl+OKp+qISL+OSIeiog7I6KxsCclSapLJoiSJPVe4zZDTN9fcd8LmTkBuAb4Sjn2VeDGzJwIfBtYUI4vAH6SmUcCbwUeKsfHAtdm5luAjcD7qvx8JEnqIjKz6DZIkjQoRMSLmbl3N/EngXdm5uMRMRT478zcLyI2AAdk5uZy/OnMfENErAdGZeZLFec4BPhRZo4tb18MDM3Mv6v+M5MkqcQeREmS+kf2cHtnvFRxewvWCpAkDTATREmS+sf7K65/Ub79c+Cs8u0PAv9Zvn0X8CmAiGiIiBED1UhJkrbHXyYlSeq9xohYWbH9w8zsXOpin4hYRakX8Oxy7LPA9RExG1gPfLQc/0vguoj4OKWewk8BT1e99ZIk7YBzECVJ6qPyHMSWzNxQdFskSeoLh5hKkiRJkgB7ECVJkiRJZfYgSpIkSZIAE0RJkiRJUpkJoiRJkiQJMEGUJEmSJJWZIEqSJEmSABNESZIkSVLZ/w+9duoXvCIJaAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n",
    "我的结果显示，加了dropout的训练准确率有所下降，但是验证准确率上升，可以看到缩小了两者之间的gap， 所以说是dropout充当了regularization的作用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
